Fluctuation and Order of Antiferromagnetism induced by Paramagnetic 
Pair-Breaking in Superconducting Vortex Lattice 

Kazushi Aoyama^'^ and Ryusuke Ikeda^ 

^ Young Researcher Development Center, Kyoto University, Kyoto 606-8317, Japan 

^ Department of Physics, Kyoto University, Kyoto 606-8502, Japan 

Effects of the strong Pauli-paramagnetic pair-breaking (PPB) on the vortex lattice in d-wave 
superconductors are theoretically studied by putting emphasis on consequences of the PPB-induced 
antiferromagnetic (AFM) ordering in the spatial modulation in the vortex lattice. It is shown that 
the PPB-induced AFM fluctuation in the superconducting state leads to an enhancement of the 
f^ , vortex lattice form factor which is a measure of spatial variations of the internal magnetic field and 

^SJ ' that the enhancement becomes more remarkable as an AFM instability is approached. It is also 

demonstrated that the PPB-induced AFM ordering is assisted by the vortex-lattice modulation, and 
thus, that the resulting AFM order is spatially modulated, while it is not localized in the vortex cores 
but coexistent with the nonvanishing superconducting order parameter. These results are discussed 
r^^. ' in connection with two phenomena observed in CeCoIns, the anomalous field dependence of the 

^\] I vortex lattice form factor and the AFM order appearing inside the high-field and low-temperature 

superconducting phase. 
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I. INTRODUCTION 
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O ■ An antiferromagnetic (AFM) instability occurring near a superconducting (SC) phase with d-wave symmetry has 

^ \ been repeatedly detected in experiments and discussed theoreticalh;. In zero field, the AFM and SC orders are 

"^ ■ competitive with each other and tend not to coexist with each other [1| . Recently, AFM quantum critical phenomena 

near the SC pair-breaking field Hc2{0) have been observed commonly in the heavy- fermion superconductors such as 

CeCoIus iJlJ], pressured CeRhlus ^], NpPd5Al2[6|, and Ce2PdIn8 J,], which indicates field-tuned proximity of an 

AFM instability to the onset of the SC order. However, these materials do not show any magnetic order in the normal 

1-^ state: An AFM fluctuation is merely enhanced with decreasing field toward -ffc2(0) in the nonmagnetic normal state. 

P^ • Further, it has been also clarified in CeCoIus that the AFM fiuctuation is enhanced as -ffc2(0) is approached by 

O ! increasing the field in the SC phase. A basic origin of the strong AFM fluctuation induced in the vicinity of Hc2 (0) 

,^ , • seems to consist in characteristic features of these superconductors, i.e., the strong Pauli-paramagnetic pair-breaking 

I (PPB) effect and a d-wave SC pairing symmetry. In our previous paper [8|, we have shown that, in a SC phase with 

y—i . a d-wave pairing symmetry, the PPB effect enhanced by increasing field and decreasing temperature tends to induce 

J> ■ an AFM order and that the field-induced AFM fluctuation found below Hc2 (0) can be explained as a result of this 

C , novel PPB effect. In this paper, we will extend our previous theory to describe spatial variations of the AFM order 

j^ ' in a SC vortex lattice by bearing the experimental studies on CeCoIns in our mind. 

CeCoIus is a spin-singlet superconductor with a layered crystal structure along c-axis [9|] and its SC pairing symmetry 

is believed to be of d,j;2_y2-type [101 ■ ^^ the SC state in a magnetic field perpendicular to the basal plane (H || c), 

^^ " an anomalous field dependence of the vortex lattice form factor (VLFF) , which is a measure of spatial distribution of 

the internal magnetic field in vortex lattice states, has been observed in neutron scattering experiments. It has been 

known that, although VLFF decreases with increasing field in the conventional type II superconductors [III, [12], it 

rather increases with increasing field in CeCoIns, and that the slope of VLFF in this material becomes sharp abruptly 

in higher fields near Hc2{0) at low temperatures |13lll4|. First, this intriguing behavior has been explained as a result 

of strong PPB [15| . However, the origin of the abrupt increase in higher fields has not been sufficiently explained 

}J] ■ there. In the preceding theoretical study on VLFF in Ref. [l^, the strong AFM fluctuation induced in the SC state 

Cd I just below the Hc2{T) curve [3, |j, [l8l - l2G | is not taken into account. If the AFM order and fluctuation occurring in 

CeCoIus just below Hc2{Q) are of a SC origin, it is rather necessary to incorporate effects of the PPB-induced AFM 

fluctuation in explaining VLFF data. In addition, a possible relation between such a fleld-induced increase of VLFF 

and the proposed FFLO state [la, [l3l at the high field end of the SC phase should be considered. 

On the other hand, in a magnetic field parallel to the basal plane (H || ah), the existence of a high-field and 

low-temperature (HFLT) SC phase has been clarified previously and has been identified with a spatially modulated 

Fulde-Ferrell-Larkin-Ovchinnikov (FFLO) [2l|, [22| vortex lattice state |23,[2J1. However, recent neutron scattering 

experiments have shown that an AFM order with its staggered moment vector oriented along c-axis exists only inside 

the HFLT phase [25, 26). On the other hand, the doping experiment [27i] indicating that the HFLT phase is quite 

sensitive to both the magnetic and nonmagnetic impurities and is destroyed by quite a small amount of nonmagnetic 

impurities supports the FFLO picture on this HFLT phase [2^. Further, a recent NMR measurement has clarified 

the presence in the HFLT phase of normal state regions possibly corresponding to the FFLO nodal planes of the SC 



order parameter, while the AFM order in the HFLT phase is, at least in the higher field region of the HFLT phase, 
apparently homogeneous in real space |29|. Although several microscopic pictures on an AFM order in the d-wave 
superconductors in high fields have been proposed so far [30l - l32l |. these experimental results are comprehensively 



explained by the scenario that the HFLT phase is a realization of the longitudinal FFLO vortex lattice state [28|, and 
that the AFM order observed inside the HFLT phase is induced by PPB and favors coexistence with the SC order [30| . 
In Ref. [201 5 the spatial distributions of the PPB-induced AFM order in the presence of the FFLO modulation parallel 
to the applied magnetic field has been theoretically investigated in the Pauli limit where the effect of the in-plane 
vortex-lattice modulation is neglected. Although the result obtained in the Pauli limit seems to give a correct picture 
on spatial orderings over larger scales, the vortex lattice modulation must be inevitably included to describe local 
properties such as the internal magnetic field and the spatial distribution of the AFM order. In this paper, we will 
investigate the spatial distributions of the internal magnetic field brought by the AFM fluctuation and a possible AFM 
order in the presence of the vortex lattice, taking account of both the orbital and Pauli-paramagnetic pair-breaking 
effects in the Ginzburg-Landau (GL) approach where the perturbative expansion with respect to the SC and AFM 
order parameters is used. It will be shown that VLFF is enhanced by an additional magnetic screening brought by 
the PPB-induced AFM fluctuation and that the enhancement becomes more remarkable as an AFM instability is 
approached, which suggests that the anomalous field dependence of VLFF observed in CeCoIns in H || c is due to the 
PPB-induced AFM critical fluctuation. It will also be discussed that the spatial modulation of the SC vortex lattice 
enhances the AFM fluctuation, and thus that an AFM order with a spatial modulation synchronized with the vortex 
lattice appears. The modulated AFM is not localized in the vortex core but prefers to coexist with a nonvanishing 
SC order, which is the same tendency as that of the AFM order modulated by the longitudinal FFLO structure of 
the SC order parameter [sl. l30j. 

The outline of this paper is as follows: In Sec. II, we introduce the theoretical model and derive a Ginzburg-Landau 
(GL) free-energy functional and a Maxwell equation for the internal magnetic field. The effect of the PPB-induced 
AFM fluctuation on VLFF in the case with H || c is discussed in Sec. III. This is followed by Sec. IV in which we 
argue the PPB-induced AFM order in the SC vortex lattice state in the case with H || ab. Summary is provided in 
Sec. VI. 

II. FORMULATION 

A. Electronic Hamiltonian 

We start from the electronic Hamiltonian involving a kinetic energy for nonintcracting quasiparticles with the 
Zeeman energy Hq and two interaction terms leading to d-wave superconductivity Hsc a-nd antiferromagnetism 
'Hafm- It can be written as H = 'Ho + "Hsc + "Hafm, where 

no = Y. f fUr)\s(-iW + \e\A{r))-afiBgB{v)\^,{r), 

„ Jr I ^ -I 

^SC = -|.g|^#^(q)*(q), 
q 

Hafm = -C/^St(q).S(q) (1) 

q 

with 



^ P 



S(q) == Y "^p,a('^)«,/3Cp+Qo+q„^- (2) 

Here, the layered crystal structure with an interlayer distance d in the z direction is assumed, and a position of a 
quasiparticle r is defined by {r±,dj) with an integer j specifying the location of the layer. The unit fi = c — k^ — I is 
used throughout this paper. In the Hamiltonian, Cp^a is the annihilation operator for a quasiparticle with momentum 
p and spin projection a, e(p) is a kinetic energy measured from the Fermi level Ep, and the Zeeman energy is 



expressed as fj,B gB{r) with the magnetic flux B(r), a g-factor, and the Bohr magneton /ie- The vector potential A(r) 
is expressed as A(r) = Ao(r) + a(r) and is related to the magnetic flux by the equation 

B(r) = Vx A(r) =iJZ + V xa(r), (3) 

where V x Ao(r) = H, and spatially varying internal magnetic fields are given in terms of a(r). The direction of the 
uniform external magnetic field H is denoted by Z and will be fixed along the z axis {Z = z) in H || c case and the y 
axis (Z = y) in H II a6 case. Concerning the interaction terms, 1^1 and U are coupling constants with positive values, 
(Ti (i = x,y,z) are the Pauli matrices, Wp denotes a SC pairing symmetry, and Qo — (7r/a, 7r/a, Tr/d) with a lattice 
constant a in the ab plane is the commensurate nesting vector. Since the pairing symmetry Wp is assumed to be of 
dx2_y2-tjpe, the identity 

W^P+Qn = -Wp (4) 

is satisfied. In our model Hamiltonian, antiferromagnetism is induced by the nesting property of the dispersion 

e(p + Qo) = -e(p) + r,(5ic, (5) 

where the deviation from the perfect nesting condition is scaled by the SC transition temperature Tc and is measured 
by the dimensionless parameter Sic !33!|. Although Sic is p-dependent in general. Sic is assumed to be a constant 
value since the details of the dispersion do not change our result qualitatively [30l . Then, the Fermi velocity vector 
Vp defined by de{p)/dp satisfies the relation Vp+q^ = — Vp. 

In order to discuss AFM order and fluctuation in the SC state, we introduce the SC pair-field A(q) and the AFM 
staggered field ni(q) which are defined by 

A(q) = |.g|(*(q)), 

m(q) = ;7(S(q)), (6) 

where () represents the statistical average. Here, A(q) (m(q)) plays a role of the SC (AFM) order parameter. In the 
mean-field approximation, the interaction terms in the Hamiltonian are expressed as 

^SC = ^5]|A(q)p-5](A(q)*t(q)+H.c.), 

-Hafm = ^5]|m(q)p-^(m(q).St(q)+H.c.). (7) 

q q 

Although, in principle, the AFM moment vector m(q) can be oriented in any direction, the two typical configurations, 
m II H and m _L H, will be considered since, in CeCoIn5 of our interest, the configuration m || c _L H has been 
confirmed in the AFM order observed inside the HFLT phase in H || a6 case and m || H would be realized in H || c 
case if m is locked in the c-axis. 

B. Ginzburg-Landau free energy functional 

The free energy of this system is given by F = -nnTrc^t,A.A*,m(exp [ — ("Ho + "Hsc + 'Hafm)/T]) + 
J d'^rB^(r)/(87r). In this paper, we derive the Ginzburg-Landau (GL) free energy of a form expanded in powers 
of both |A(q)| and |TO(q)|, taking both the orbital and Pauli-paramagnetic pair-breaking effects into account. For- 
mally, the mean-field GL free-energy density in the present system can be written as 



^gl(A, m) = ^f ) + ^W + 4^' + 4?' + -^fi + :f!^\ (8) 

where J^^ [J^m] is the ^th order term dependent only on |A| (|m|), and the leading order SC-AFM coupling term 
•^A m ' which is proportional to |Ap \ra\'^ , is incorporated. We note that, although the correction to the |7tt,|^ term 
•^A m ' which is proportional to | A p | m | '' , should be also incorporated in examining the character of the AFM transition, 
•^A m is omitted here since we have already checked that it tends to make the AFM transition a continuous one [3^ . 
First, we determine the Hc2{T) curve and the amplitude of the spatially averaged gap function A^ = (|A(r)p)sp from 



the SC part of the GL free energy T^ ' + J^^ ' + J^^ ' . This procedure in which A and m are separately considered is 
justified at least near the second order AFM transition. Since J^qi^{A, m — 0) takes the form 

J-gl(A,0) = V2\A\^ + ^|A|4 + -||Ar (9) 

with the coefficients Vi whose expressions reflect the microscopic details, the discontinuous Hc2{T) curve is determined 
by 

and the A which minimizes J-gl(A, m = 0) is determined by 



\^\^ = --Yi±JSE^, (u) 

ZVe, 



Next, AFM fluctuation inside the SC phase determined by J^gl(^, 0) will be discussed based on the usual GL theory 

r-(2,2) 



(1\ (2 21 (4) 

for the AFM part of the free energy J^m + J^\ ' „, + J^m ■ The AFM instability is determined by 



;^(^i^^+-^El)-0, (12) 



where m^ = {\m(r)\'^) sp is the spatially averaged value of the AFM order parameter. The m minimizing J"gl(Ao, m) — 
Jx!l(Ao,0) is determined by 

To derive the GL free-energy functional -Fch, we will carry out the Feynman-diagrammatic calculation. Regarding 
the field dependence of the quasiparticle Green's function —{Tr(pa{Tc,T)ip1{r',0)) = Tj^e ^e„,o-(r, r') ^"^"^ with a 
fermion Matsubara frequency £„ = TTT{2n + 1), we will use the quasi-classical approximation for the Green's function 
defined in the normal state t/^^ cr(r, ri), namely, 

g,„,.(r,ri)^^e„,.(r-ri)e'l^ir'^^-^(^\ (14) 

where Ge„,a{'>^ ~ ri) is defined in the uniform normal state and its Fourier transformation is given by 

Ge^Ap) = {^£n - [e(p) - ^TTc])"' (15) 

with I = fiB gH/Tc. The orbital pair-breaking effect, which is brought by exp[i|e| J ds ■ A(s)] in Eq. (|14p . can be fully 
incorporated in the diagrammatic calculation by using the relation [34| 



(i2\e\ r ds- A(s)) A(ri) = (.-iir-nyniv) ^^^^ (^g) 



/•ri 

exp 



with 

n(r) = -iV + 2|e|A(r). (17) 

The quadratic, quartic, and sixth order terms with respect to |A| have been already derived elsewhere [23,|33 and 
are written as 

■""^F/^-M(^-*?'m)^w- 

*'f(n) = |XIEl'"pl'5...,.(p)ff-...->(-P + n). 

^A^ = 2V/^A'(n.)A*(si)A(s2)A*(s3)A(s4)|^^^^, 




FIG. 1: Feynman diagrams describing the second order terms with respect to the AFM order parameter J7Ti| in the GL free 
energy. The |mp term in the normal state J>„ is described by (a) and the correction term brought by the SC order T}^'^ 
consists of the contributions coming from the two types of diagrams (b) and (c). A solid line denotes the quasiparticle Green's 
function defined in the normal state. 



T 



kf{ii,) = -^^ |«;p|4ge„,,(p)g_e„,-.(-P + nl)g_,„,_,(-p + n2)ge„,.(p + n^-n2 



r(6) 



Z^(6)^ 



•^r = ^y ifX^(n,;)A*(si)A(s2)A*(s3)A(s4)A*(s5)A(s6)|^^^^, 

xg,„,,(p - nl + Ha) a-e„,-.(-P + n| - n2 + n^) Qe„A\* - ng + n^). 



(18) 



The concrete expressions of the coefficients Vi in Eq. ([9]) are given in Appendix IT The AFM contributions in the GT 
free energy are formally written as 



•^?^ - EG + ^^'^(q))Mq)i'' 






^m = 9 51 ^m'(qO'Ti(qi)m*(q2)m(q3)m*(q4) 



(19) 



q. (i=l-4) 



The |mp terms are described by Feynman diagrams shown in Fig. [T] Figure [T] (a) corresponds to J-m\ and J-^ 



-(2,2) 



(2)/ 



consists of the contributions described by the two types of diagrams (b) and (c) in Fig. [T] Then, i^m (q) and 
-^A.m (q^ q') are given by 



1 
U 



1 1 

c7 + 2. 



^^?'(q) = (77 + ^E^EE^-'^-(p)^-"^^(p + Qo + q^ 



I Sp 



«(o)(.4---s[^4i:„E,x.t>/. 



kSIm) = -\Y.TY.T.hL 



e„>0'-' ' a Se„=±i "'+,q 

2 



2 kp|^ es„,.(p) QeMv - Qo - q) Qe^Av - (q - q')) ^-.„,-a(-P + Hi + (q - q')) 



with 



wpWp+Qo ^£„,<t(p) ^e,.,?(p - Qo - q) G-e^-A-v + Qo + q + ni) 5_^^ _^(-p + ni + (q - q')) 



,g-(q-q')A*(r)A(Sl)|,,^r 

-1 5] 27rriV(0) ^(z ., J yj^ e' "■•('i-'i') A* (r)^2 



1 



d^^ d(-) 



7(-) 






WpWp + Qo 



'(T,ni cr,ni+g— g' +,9 
1 



7(-) 



.,dlidi+i di+^ rfiidi+) 



^o-.IIi+g-g' "+,9 "+,«' ^'.IIi "+,g +.,q' 



A(si) 



FS 



(20) 



^(±) 



c'^n. = 2i e„ + 2a/T, ± vp • n, 



TABLE I: Coefficients Pj, Aj, Bj, and Cj in Eqs. ^ and ((30] 



J Pj A, 



B, 



C, 



1 2 pi + P2 p2 + p3 -P2 

2 2 pi P3 p2 

3 -1 pi pi + P3 -pi - P2 
4—1 pi -p3 P2 



d^*] - 2i£„ + (a+^)/r,±(T,5ic+Vp-q), 



(21) 



where s^^^ represents sgn(e„), the summation for the momentum p has been carried out by using the usual replacement 
J2r, ~^ ^(0) / ^^(p) Ofs' -^('^) ^® ^^^ density of state per spin at the Fermi level, ()pg represents the angle average on 
the Fermi surface, and the approximation e(p + 11) ~ e(p) + Vp • 11 (|n|/|p| ^ 1) is used. Tjv is the AFM transition 
temperature in the normal state. It should be noted that W changes its sign depending on the orientation of m relative 
to the external magnetic field H, i.e., a — a for m || H and W = —a for m _L H. 
By using the transformation 



1 

15 



dp exp 



Dp (Rei:)>0), 



Eq. ((20l) can be rewritten as 

-+i.(?)(q)^A.(0)(ln- + y^ 



dp 



2Tlt 



sinh(27rtp) 



/cos(p,0)(cos((JiC+T-Vp.q)p 



FS 



(22) 



(23) 



and 



k'i^sm) - ^iiT dp' j e-^---^^*[r)\ y: tp^ho.{j:p..A, 



x( |Wp| exp 



«Se„((5ic(i?j + Cj 






B, 






C, 



exp \iSeT^ Vp • III Aj A(s 



FS 



SI— >r 



where the relation Wp+Qo = — Wp is used, the functions 

2TTt 



fcosix,y) 
fsin{x,y) 



■ cos I / 



2yH (x~y) 



(25) 



sinh [27ri x] 

27ri . /-F To cr + o", -IN 

— -=-— ^sm(/ 2y+ (x-y) 1 

smh [27ri a:J V L a J / 

are introduced for convenience, and the coefficients Pj, Aj, Bj, and Cj are shown in Table. HI In the same manner. 
Km i^i) in Eq. p^ is calculated as 

K^r^Hqr) = T ^ ^ e.„,^(p) ^s„,f(p + Qo + q2) Ge^AP + q2 - qs) ^^^.^(p + Qo + qi ) fe, 

e„,o- p 

iVfO) ^ /"^ ^ / ^ 

= &, ^^;5- n / dP^ fcosiY^ Pt, 0) COS ((5ic ( ^ Pt) 



x/ cos(t^ ^Ivp •qi(pi +P2) +Vp •q2(-/32 + /03) + Vp • q3P2 
+ COS (t^^ Vp • qipi + Vp ■q_2{-pi + P2) + Vp • q^ipi + Ps 



(26) 



FS 



where fe, denotes (J^i.gi-qa+g.,- 

Next, we calculate f^ e*'"^'*^'* )A*(r) exp [is^AT^^Vp ■ Hi] A{si)\s-^r in Eq. (p4|) . In the presence of a magnetic 
field, the SC gap function A(r) has a spatial modulation due to vortices induced by the orbital pair-breaking effect. 
In this paper, the SC gap function is assumed to take the form of a familiar Abrikosov vortex lattice 



A(r)=A^o(^,>^) 



(27) 



with 



Vo[X,Y) 




Y. ^^p 



S — — CXD 



\rH 



-(- 
2\rH 



sk 



(28) 



where th is the magnetic length defined by r^ = {2\e\H)^^^^ and a structure of the vortex lattice is specified by k. 
The coordinate (X, Y) denotes that in the plane perpendicular to the magnetic field and is defined by {x, y) for Z \\ z 
and {'^^^^z,^~^^'^x) for Z \\ y, where 7 denotes the ratio of the SC coherence length in the basal (ab) plane to that in 
the direction along c-axis and is expressed as \/{Vp 2;)Fs/(Wp 2)fs- In our numerical calculation, k = v7r-\/3 describing 

the triangular lattice is used since, in the high field SC phase of CeCoIns in H || c, the square lattice characteristic of 
d-wave superconductors is deformed into nearly triangular ones due to the strong PPB effect [10, [3 • ^Y using the 
identity [ 23l | 



exp 



a^--n 



Tc 



MX,Y) 



exp 



1 



A'{\v\ 



V 



*2^ 



^Q 



{X + A7^*V2rH,Y) 



(29) 



with 77 = (wp.x + ivp^y)/(^y/2rHTc) for Z \\ z and 77 = (7^/^Up,z +^7 "^^^Wp^j;)/(%/2rH Tc) for Z \\ y, we obtain 

12 3 






(2) ^.K.,X,y) ^ p^ ^_( ^ ^^^ ^^.) ^^^ (^^^ (5^. ^ c,) 

1=1 



J = l 



x/lwppexp [ 



9 ^J 



'(r; X 'Ki_rH)z 



- C^^-' - ¥-^ - ^-') "- (^^^^^^r^-^') ) <f' 



where ry = (Re 77, Im 77) and Kj^ = {Kx,Ky) is a reciprocal lattice vector of the SC vortex lattice in the plane 
perpendicular to the magnetic field. The details of the calculation in incorporating the orbital pair-breaking effect 
and the useful identities which will be used below are shown in Appendix I. 

According to Eq. (PO)) . the SC-AFM coupling term is nonvanishing only when (q^ — 9^, % — <l'y, Qz — Qz) is equal to 
-iK,,Ky,0) in Z II z and -i-,-^/^Ky,0,^^/^K,) in Z \\ y. 



C. Vortex lattice form factor and the internal magnetic field 



In the vortex lattice state, the SC gap function is expressed as |A(Ar, y)p = ^j^ T^ exp 
the longitudinal flux distribution B{X,Y) Z is also expressed with Kj^, 

BiX, Y)=J2 F-K^ exp [i Ki • (X, Y) 



iK^- iX,Y] 



Then, 



(31) 



The Fourier component IFk^I with the smallest |Kj^| 7^ corresponds to VLFF which is, in the neutron scattering 
experiment, obtained from the integrated intensity of a diffraction peak [13| . Since VLFF measures the inhomogeneous 
part of the magnetic flux, the problem results in obtaining the spatially varying internal magnetic field. 

We derive a Maxwell equation relating the internal magnetic field b(r) = V x a(r) in Eq. ^ to screening currents 
from the saddle point equation of the total free energy with respect to A SF/SA = 0, i.e., (^S'Hq/SA')\._. + V x 
(V X A(r))/(47r) = 0. Then, the Maxwell equation is given by 



^V X B(r) ^Tj2J2i- |e|vp)Ge„^,(p; r) - V x (t ^ ^ /iBgaG,„..(p; r)) Z, 



(32) 



€n,(7 p 



€n,a- p 



where higher order terms in A have been dropped in Eq. (j32p because their contributions are negligibly small within 
the quasi-classical approximation, and Ge,^,cr(p;r) is a Fourier transformation of the quasiparticle Green's function 
Gg^^ai^, r') and will be given later. In the right-hand side of Eq. (|5 ^, th e first term is the usual term expressing the 
current density, while the second term arises from the Zeeman term [30]. In this paper, we will use the perturbative 
expansion for the quasiparticle Green's function with respect to both |A| and \m\. Then, Gj„.cr(p;r) can be written 
as 



G,„,.(p; r) = G(2) (p; r) + G^, .(p; r) + G^"-'} (p; r) + G^"^ (p; r) 



(33) 




FIG. 2: Feynman diagrams for the SC and AFM contributions to tiie internal magnetic field. Here, the diagrams (a), (b), (c), 
and (d)-(f) denote the contributions proportional to \Af , lA]*, |mp, and |Ap|mt'^, respectively. For the lAI'^lm]^ terms, not 
all the diagrams but only typical ones are shown: a diagram with two wavy lines on a single solid line (d), one with two wavy 
lines separated by the vertex (e), and one with two wavy lines separated by the pair field |A[ (f). The two diagrams (d) and 
(e), and the diagram (f) denote the internal fields arising from the free energies described by the diagram (b) in Fig. [T]and the 
diagram (c) in Fig. [TJ respectively. 



with 



G(4j,,(P; r) = Ge^A-P + ni - Ul) G-e^^-AP + ^l) Ge„A-P) G-e„,-a{P + ^s) Ge„A~P ~ ^3 + ^l) 

x|«;p|4A(si)A*(s2)A(s3)A*(s4)|^^^^, 

G(oj(p; r) = ^ m(q) ™*(q') e^i^-^'>'' Ge^A'P - Q') Ge„A~P - Qo) Ge,.A-P - q)> 

q,q' 
G(2J(p; r) = - ^ m(q) m*(q') e'Ci-i')- g^^^^^^-p + U, q') Ge^A'P + n| - q) 
q,q' 

X \wp\^{Ger.A-P - Qo + Hi) Ge„A-P + ni - q) G-e^^-aip + ^) + (Hi O H^, q O q')} 
+ kp|'{e-s„,-a(p + q') G-e^MP + Qo) G^e,.,^a{p + q) 

+Ger.A-P + Qo + Hi) e-e„,-?(P - Qo) Ger.A-P + Qo + ^l)} 

-wpWp+Q„ \^g,^A-p + Qo + Hi) a-e„,-?(p - Qo) G-e„,-Ap - q') + (ni <-> n|, q ^^ q')} 

xA(si)A*(s2)L ^,, 



(34) 



where GeZ\<j and GJl'J denote the contributions proportional to |A|" and |A|"|mp, respectively. The contribution 
GeJ,a has been dropped since it only gives a spatially uniform flux. We note that the SC fourth order term G*^*^ 
is incorporated since, in the case with strong PPB of our interest, |A| is finite even on the Hc2{T) curve at low 
temperatures due to the first order Hc2 transition and higher order SC contributions are not safely negligible. The 
contributions G^^^ G^^\ and G^"'^^ are described by the Feynman diagrams (a), (b), and (c) in Fig. [21 respectively. 
The SC-AFM coupling term G*^^'^^ consists of the contributions described by the three types of Feynman diagrams 
(d)-(f)inFig.[l^ 
Equation (j32]) indicates that the magnetic flux can be written as 



B(r) =HZ + b^^)(r) + bi^'(r) + b(^^^)(r) + b^^f (r), 



(35) 



where by(r) and b]^'™^(r) are the internal magnetic fields coming from Ge,l,a and G^^™ , respectively. First, the 
second order contribution b)^ (r) will be calculated. By using the transformation (|22|) . we have 

r, --27ri Sp 



T^/d£(p)G(S(p;r)=T^ 



dit d'^\ 



A(si)A*(s2) 



= 7— / dpidp2 — 



27ri 



X I Wp I ' exp 



'^c Jo 

|2 



sinh [27ri(pi + P2)] 2 



- ^ (zse„) exp ise„12a{pi+ P2) 



=±1 



iSe„T-i(piVp • Hi + p2Vp • n^) A(si)A*(s2) 



(36) 



Further, with the help of Eq. (|65|) . we obtam 



bi^^(r) 
47r 



Z 



2|e|t;FrHA^(0)|A 



2 /-oo 



/•OO 2 



+P2)' 



Ki^O 



FS 



f(2) , X fq-lf^^TH, 

^k1,Vp(P1,2) = C0S(^ -= {Pl-P2 



/cos(Pl +P2,Pi+P2) ;7^ ;:^sinh( (/O1+P2) 



vplKrHl^ 



V2 



'--^/sin(Pl +P2,Pl +P2JCOSh( -= [pi + P2) 



(37) 



where vp is the Fermi velocity in the pure two dimensional system and ^o is the SC coherence length at T = defined 
by Co = vp/(2'kTc). Here, the term proportional to (i|e|vp) is transformed, with the rotation operator V x • • •, in the 
form 



E .Vp/K.(vp)e'^-(^'^) 



V X 



E ^i|^/K.(vp)e--(-'-))i, 



K±^0 



K 



(38) 



where K is defined by K = {K^,Ky,0) for Z || z and K = {-f-^/^Ky,0,j^/'^K^) for Z \\ y. 

To check the result obtained here, we consider the internal field near Tc at nearly zero magnetic field, i.e., in the GL 
region. We take the limit t ^ 1 and i7 ^ (or equivalently, 77 — )■ 0) in Eq. (I57|) in H || c case. Then, the contribution 
arising from the Zeeman term, which is proportional to / in Eq. (I37[) . vanishes, and the remnant orbital contribution 
is expressed as 



lim 



-bi^'(r) 



2^|e|4iV(0)|A| 
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dp F =r 

sinh [2ttpi + P2J 



: (vpXK^)^ 



FS 



E^^ 



(2) giKi-r^ 



K, 






(39) 



FS 



with 



^^" = (2^)2 8^^^^^' 



(40) 



where /„ (i/3p2/sinh[27rp] = 4^jj>g l/[27r(2n + 1)]^ is used. Equation ([M)) is a well-known familiar form of the 
internal magnetic field obtained in the GL region |34| . 

Next, we will calculate the fourth order contribution b^ (r). By carrying out the integral f de{p) and using the 
transformation (|22]) . we have 

1 



tJ2 I de{p)Gi%ip;r) =Tj2i-2^^s, 



^^ni cr,ni f^^na ^.n] 



4+1I d^+\ rf(+\ d^+\. t 4+ii 4+1I rf*+\ rf'+^ t 



A(S1)A*(S2)A(S3)A*(S4)| 



1 1 
T32 



E (^■^-) n / '^p- 



3 

xE i^pi^'exp 



_i ^0 



i=l 



27ri 



■ exp 



jmJ2p^ 



sinh[27ri(X;-=iP»)] 
i Se^T^M Vp • Hi Oj + Vp • n^ I3j + Vp • ria 7^- + Vp • IlJ Sj 



A(S1)A*(S2)A(S3)A* (84)1^44,1) 



TABLE II: Coefficients Qj, Pj, jj, Sj in Eqs. (|iT| and (gH) 



J Oj 



ft 



7j 



1 pi P2 P3 P4 

2 pi P2 + P3 -P3 P3 + P4 

3 pi + P2 — P2 P2 + P3 Pi 
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where the coefficients Uj, Pj, 7^, and (5j are shown in table [TTl With the combined use of Eqs. (pS)) and ([55]) . we 
obtain 
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(42) 



4=1 1=1 



with 



Concerning the internal magnetic field induced by the AFM order, the leading order |mp term which does not 
include |A| is straightforwardly calculated as 

b^''^(r) _ ^ Stt rll '■°° 



4:TtCgl 



^777^~§:~ I '^P^ ^^2 /sin(pi + P2, 0) cos ( (5ic(pi + P2) ) 

ij ^ m(q) m*(q') e^tq-q')-/ ^^s (r-^Vp • qpi + T-^Vp • q' p-. 



(44) 



FS 



.(0'2)/ 



We note that, when the AFM moment is perpendicular to a magnetic field (a — —a)^ bm' (r) vanishes since 
/,i„(x,0) = Oinm±H. 

The internal magnetic field brought by the SC-AFM coupling b^'^(r) can be derived in the same manner as that 
used in obtaining b)^'^(r). T^^ J(ie(p) G'e„',CT(p; i") is calculated as 

T^ /'d£(p)G(2J(p;r) = -27rr^(zs,j5^m(q)m*(q')e^('>-'>')"- 
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TABLE III: Coefficients pj , aj , bj , Cj , dj in Eqs. (|45)) and (|46|l 



J P] flj 



d, 



1 2 — p2 pi + P2 P4 P2 + P3 

2 2 pi + p3 p2 P3 + P4 — P3 

3 2 pi p2 P4 P3 

4 1 pi p2 + P3 — P3 — P4 P3 

5 1 Pi + P3 P2 P3 -p3 - P4 

6 1 pi P2 — p3 -P4 

7 1 —pi — p2 P2 P3 p2 + pi 

8 1 P2 -pl — P2 P2 + P3 Pi 

9 1 -pi -p2 P3 P4 

10 -2 —pi - p2 P2 P3 P2 - p4 

11 -2 — P2 Pl + P2 P3 P2 - P4 

12 —2 pl p2 Pl + P3 — Pl — P4 
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where Cj = aj + bj and the coefficients pj , a^ , bj , Cj , and dj are shown in table IIIIl Then, we obtain 
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(46) 



where the relation Wp+Qg = Wp is used and K' is defined by K' = {K^ + qx ^ Ix^^v + 1y ^ Qyi^) for Z \\ z and 
K' = ij-'/^Ky + q,- g;, 0, ^^/'Kx + 9. - ^D for Z \\ y. 

In this paper, we use the following form of the reciprocal lattice vector: 



K_irH = mi {n/k, 0) + m2 (0, k) 



(47) 



with integers mi and 7712- Then, the summation for Kj^ is replaced with ^^^ ^ , and T^ in Eq. (j37l) and Tj^ in 
Eq. (|1^ are concretely given as follows: 



^kI = <5,„i+„i2^cvcnCos f-m2(mi + 7712) ) exp 



1 /(mi7r)^ 
4V fc2 
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exp 
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8V P 
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(48) 
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Since the triangular lattice k — \/t:^/3 is assumed, there are six (7711,7712) combinations with the shortest |Kj^| ^ 0, 
±(1,1), ±(1,— 1), and ±(2,0), so that the equality |-F±i,i| = |^i,±i| = |-F±2,o| is satisfied in an isotropic system. 
However, in the present system, an asymmetry occurs in the six |-F'mi,m2l components because of the anisotropy 
originating from the d-wave pairing symmetry or the layered crystal structure. As a result, the equality |-F±2,o| = 
l-Fi^iil is not satisfied any longer, while the equality |-F±i,i| = |fi.±i| is still satisfied. Since the four (7771,7712) points 
with the same |-F'mi,m2l value, ±(1,1) and ±(1,— 1), correspond to the spots with the strongest neutron scattering 
intensity in the experiments, we will calculate |i^i,i| which is given by 



Fii = l 

1,1 y 



bi^) (r) + hf (r) + b(^^^) (r) + b^^;^ (r)! exp [ - 7 (vr/fc, fc) • (X, F )! . (49) 



The form factor |Fi_i | is easily obtained by using Eqs. (f37| . (|42l) . (J44l) . and (|46l) since b^' and h^^^ in these equations 
are already expanded with the reciprocal lattice vector Kj^. In the results of our calculation below, l-Fi.il is normalized 
by AttCgl- 

In the present theory, the layered crystal structure is reflected in the anisotropy of the Fermi velocity vector Vp. 
Although the expression of Vp should be derived from the realistic dispersion, for brevity we use the following simplified 
form of Vp : 

\'p = vf[1 — {J/Ef){1 ~ cos{pzd))] {x cos<j) + y sincj)) + Jdsm{pzd)z, (50) 

where J is an interlayer coupling constant. The Fermi velocity vector introduced here corresponds to that of the 
corrugated cylindrical Fermi surface. Then, the anisotropy of the SC coherence length 7 can be expressed as 7 = 
2-y/l — J /Ep /{t^J /Ep) and the angle average on the Fermi surface is defined by 

In the numerical calculation below, J jEp — 0.2 and ppd ~ tt are used, and then, 7 — 2.85 is obtained. 

In the numerical calculation, the magnetic field H is normalized by the orbital limiting field in the pure two 
dimensional system -ffjD v*-*) ~ 0-28/(|e|^g). Then, the two field-dependent energy scales fiBgH and 7;i?/(27rr//) 
which correspond to the paramagnetic pair-breaking effect and the orbital one, respectively are expressed as follows: 

— = -^^=^oV2\e\H^';;'''>(0)Vh = 0.748 Vh, (52) 

TH 27r rn T^ 

where the Maki parameter aM.j = \/2Hy'^ (0) / Hp{0) measures the strength of the paramagnetic pair-breaking effect 
and h = H/H^ ' (Q) is the normalized magnetic field. Here, Hp{0) — ttTc/ {\/2e'"^ ^^9) — ^■'^Tc/iJ.Bg is the Pauli 
limiting field at T = 0, where je — 0.577 is the Eulcr constant, while i/ (0) is the orbital limiting field at T = 
for fields parallel to the j direction. For the parameters used in our calculation. He (0) — 1.266 i72D ('-') ^^'^ 
H^l'^\0) = 4.871 7J^D^^(0) are obtained. 

III. ANTIFERROMAGNETIC FLUCTUATION IN H | c 

In this section, we consider effects of the PPB-induced AFM fiuctuation on VLFF in H || c case and discuss the 
origin of the anomalous field dependence of VLFF observed in CeCoIns [lj,[ijl. Throughout this section, aM,c = 5.8 
is used in obtaining main results. The temperature-field phase diagram obtained by minimizing J^Qi^(A,m = 0) is 
shown in the inset in Fig. [5l where the Hc2 transition at low temperatures is of first order, and a possible FFLO 
vortex lattice state appearing just below the Hc2 (T) curve as a narrow HFLT region [la, [l3| is neglected because it 
does not affect the SC properties in the field range of our interest. In fact, we will demonstrate later in Sec. IV C 
that, even if the FFLO vortex lattice is taken into account, VLFF m H || c case will be hardly affected by a FFLO 
spatial structure. Below, we will show results of our calculation in two cases, m ± H and m || H, since the direction 
of m relative to the external magnetic field H has not been confirmed in H |j c case. 
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FIG. 3: (Color online) Logarithmic plots of VLFF |-Fi,ip at t — 0.1 as a function of magnetic field scaled by the upper critical 
field at T = Hc2,c{0) for various values of Maki parameter cva/.c- Here, I.F'i.ip is normalized by (AtvCg'l)'^- Dotted, dashed, 
and solid curves correspond to the cases with q:m,c = 0.01, qm,c = 3.5, and q:m,c = 5.8, respectively. The form factor |i^i,ip is 
enhanced by strong PPB at high fields, and as a result, the slope changes its sign from negative to positive. The inset shows 
the regular plot of the |Fi,i| curves shown in the main panel, where the results for aM,c ~ 0.01 (dotted curve) and aM,c ~ 5.8 
(solid one) are measured by the left and right vertical axes, respectively. 



A. Paramagnetic pair-breaking effect on VLFF 

First, PPB effects on VLFF in the case without AFM fluctuation will be discussed. Figure [3] shows the field 
dependences of VLFF at i = 0.1 for aM,c = 0.01 (dotted curve), aM.c — 3.5 (dashed one), and aM,c = 5.8 (solid 
one). As one can see in Fig. |31 VLFF is enhanced especially at high fields by the PPB effect. For the sufficiently 
large a^/.c, VLFF increases with increasing field while it decreases in the case with the small aM.c- The inset in Fig. 
[3] shows the regular plots of VLFF as a function of magnetic field scaled by Hc2,c{0) in the two cases, aM,c = 0.01 
and aM.c = 5.8. The difference in the field dependence of VLFF can be clearly seen. To see that VLFF measures 
spatial distributions of the internal magnetic field, the longitudinal internal magnetic fields b}^-^(r) + b^'(r) in the 
cases with q;m,c = 0.01 and aM,c = 3.5 are shown in Fig. |4] (b) and (c), respectively. In the SC vortex lattice state 
shown in Fig. 2] (a) , the internal field in the direction along the applied magnetic field is induced at the vortex core 

as a result of the magnetic screening outside the vortex core. The anisotropic structure of b)^ (r) + b)^'(r) around 
the vortex core is due to the mismatch between the four-fold d-wave pairing symmetry and the six-fold triangular 
lattice symmetry. Comparing (b) with (c), one can see that, due to the PPB effect, the magnetic flux is concentrated 
inside the vortex core as a result of the enhanced magnetic screening outside the vortex core. The enhancement of 
VLFF shown in Fig. [3] is understood as a consequence of the concentration of the magnetic flux inside the vortex 
core, or equivalently, the enhanced magnetic screening outside the vortex core [15|. The increasing behavior in the 
fleld dependence of VLFF due to strong PPB has been already argued by Ichioka and Machida who discussed this 
issue quantitatively by using the quasi-classical approximation where the SC gap and both of the two pair-breaking 
effects are fully taken into account [l^ . The qualitative consistency of our result with their result indicates that the 
perturbative expansion with respect to |A| used here is valid in the discussion on VLFF. 



B. AFM fluctuation induced inside the SC phase 



Next, we will show that PPB enhanced sufficiently by increasing the external magnetic field induces AFM fluctuation 
inside the d-wave SC state. Since this novel PPB effect has been already studied thoroughly elsewhere [30] in both 
m _L H and m || H, in this subsection, we will only give a brief review of this PPB effect with an example of our 
result in the case with m _L H. 

-(2,2) 



Figure [5] shows the field dependence of the SC-AFM coupling term J'^ ' ^ for Sic = 0.005 in m _L H. Since Eq. (fT2l) 

(2 2) 

determines the AFM instability, negative values of J^^ ' „j indicate that an AFM ordering or an enhancement of AFM 

(2 2) 

fiuctuation tends to occur in the SC state. Noting the physical implication of J^^ ' ^, one can see that, due to PPB 
enhanced by decreasing temperature and increasing field, AFM fluctuation is induced in sufficiently low-temperature 
and high-field region, while, in higher-temperature and lower-field region, the AFM fluctuation is suppressed by the 
SC order. The field hcp at which the AFM fluctuation is strongest is flxed at the Hc2 transition. The jump seen at 
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FIG. 4: (Color online) Spatial structures of the SC gap |A(f)|^ (a) and the longitudinal internal magnetic fields normalized 



.(4)/ 



by 4nCGL, b^'(f) + b^'(f), for weak PPB (qm.c = 0.01) (b) and slightly strong PPB {om.c = 3.5) (c) at i = 0.1 and 
H/Hc2,c{0) = 0.96 in the case with no AFM fiuctuation. f denotes the dimensionlesss coordinate defined by f = r/rn- Due to 
the PPB effect, the magnetic fiux is concentrated inside the vortex core as a result of the enhanced magnetic screening outside 
it, which is refiected in the enhancement of VLFF shown in Fig. [S] The anisotropy of the internal field around the vortex core 
is brought by the mismatch between the d-wave pairing symmetry and the triangular lattice one 
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FIG. 5: (Color online) Field dependences of the SC-AFM coupling term in the free energy T^ ' ^ in H || c case with m _L H at 
t — 0.02 [lower (red) curve], t = 0.06 [middle (blue) one], and t = 0.1 [upper (green) one]. In obtaining the result, aM,c = 5.8 
and Sic = 0.005 are used. The Hc2{T) curve obtained is shown in the inset, where an arrow denotes the point at which the Hc2 
transition changes its nature from the second order to the first one and the FFLO vortex lattice is neglected here. To make it 
easy to grasp which region in the phase diagram is considered, the temperatures t — 0.02, t — 0.06, and t = 0.1 are indicated 
by the left (red) vertical line, the center (blue) one, and the right (green) one, respectively in the inset. PPB enhanced strongly 
by increasing field and decreasing temperature induces AFM fluctuation inside the SC state. The large energy gain in T^ ' ^ 
just below the Hc2 transition at the low temperature indicates that the AFM fluctuation gets stronger toward Hc2{0). 



the Hc2 transition is due to the finite SC energy gap | A| originating from the first order nature of the Hc2 transition. 
These results suggest that the PPB-induced AFM fluctuation gets stronger toward Hc2{0) inside the d-wave SC state 
with finite | A| and that the strong AFM fluctuation with hcp located just below the Hc2 transition at the extremely 
low temperature causes the quantum critical phenomena around 7fc2(0). It should be emphasized that the result 
obtained here is not accidental to the GL expansion but intrinsic to the present system. This novel effect of strong 
PPB has been theoretically confirmed in the Pauli limit case where jAj is fully taken into account [S^. We note 
that, in m || H, the AFM fluctuation is also induced inside the SC phase by the PPB effect (see Fig. 4 and Fig. 
9 in Ref. [SO])- As a typical example of physical phenomena caused by the PPB-induced AFM fluctuation in the 
SC vortex state, we will discuss effects of the PPB-induced AFM quantum critical fluctuation on VLFF, bearing the 
experimental data on CeCoIns in our mind. 



EfTects of AFM fiuctuation on VLFF 



Below, we will examine effects of the AFM fluctuation on VLFF. Feynman diagrams for the internal held brought 
by the AFM fluctuation are shown in Fig. [5] where a wavy line denotes the AFM fluctuation and b™' has been 
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FIG. 6: Feynman diagrams for the contribution from the AFM fluctuation to the internal magnetic field. The diagrams (a), 
(b), and (c) are obtained by connecting two wavy lines in the diagrams (d), (e), and (f) in Fig. [2] respectively. A wavy line 
represents the AFM fluctuation. 



dropped since it only gives a spatially uniform contribution to the magnetic flux. Then, the spatially varying internal 
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magnetic field is written as b^ 



-bk^^(r) 



b[iuc^(r), where 



,(2,2) 



(r) 



4^Cgl 7C(3) Co T^ 



8n r^lAp ^ (|m(q)P) ^ 

Z^ 7-2 Z^ ^i 



(2) „,:k_lt^ 



n 



12 



dp.5:p,VkJ^6p^ 



^Ki 



,(pj;q>q) 



-ia,+b,y 



J=l 



FS 



(53) 

Here, the commensurate AFM fluctuation {6ic ~ 0) is assumed for brevity. The properties of the AFM fluctua- 
tion are specified by the AFM correlation function (|m(q)p) which is introduced phenomenologically through the 
approximation (|?Ti(q)p) ~ T^^ x(q:*w„) with the familiar form of the susceptibility |38l44l| 



x(q,«'^n 



e'(/^,i)C^ 



7V(0) l+e{h,t)\q X i|2 + \c,n\/{To^-Hh,t)^l,y 



(54) 



where <^n and Fq are a length scale and an energy scale characteristic of the AFM fluctuation, respectively, ^{h, t) is 
the correlation length which diverges at an AFM instability, and the two dimensional fluctuation is assumed in Eq. 
([51)1 based on the experimental data [331 • Since, as we discussed in the previous subsection, the AFM fluctuation 
strongly enhanced at hcp gets stronger with decreasing temperature, we assume that the AFM instability is located 
at an extremely low temperature and the high field hcp- Then, the correlation length is considered to take the form 
^(h,t) = ^^{t+\l- h/hcp\)~^^'^ [Ulil- We note that the condition ^{h,t)~'^ = corresponds to Eq. ^ which 
determines the instability of the PPB-induced AFM order. Since x(q, ?w„) has a dominant contribution at q = 0, we 



will calculate the right-hand side of Eq. ((53)) by using the evaluation b\^ 
part relevant to the summation for q in Eq. (|53p is calculated as 
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Imx^(q,t 



qc£c 
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dx coth I —2- X 
\2T 



tan 






(55) 



where x^ is the retarded susceptibility and the replacement ^ 



(2^) 



■ / dqz J d4> J qdq is used. In the numerical 



calculation below, Sc = 2Tc, Fq = Ep, N{0) = pp/{2Tr vf), Qc = pp are used. We believe that the approximation 
used here, (|m(q)p) ~ TJ2uj xi'ii^^n), which corresponds to setting w„ ~ in the quasiparticle Green's function 
with an Wn- dependence, properly gives the AFM contribution since xil^ * Wn) has a dominant contribution at a;„ = 0. 
On the other hand, it is known that the cj„-dependence in the Green's function produces the imaginary part of the 
self energy of a quasiparticle described by the Feynman diagram (a) in Fig. |6l Since the imaginary part of the self 
energy, namely, the quasiparticle damping suppresses the PPB effect [23[, it cannot be neglected in examining any 
PPB effect. By carrying out the diagrammatic calculation, we can evaluate the imaginary part of the self energy near 
the Fermi surface as follows: 



lml]«(kf,0) 



4 PpC^ 



(■ 



C(M)/A£Xl2x-l/2 

L 2<o vrj. 



(56) 



with Ae — —TcSic + (ct + <j) ITc (for details, see Appendix III). In the expression ([5^ . a familiar form of the 
quasiparticle damping caused by the AFM spin fluctuation near the hot spot [40] is extended so that it includes the 
Zeeman energy shift. Noting that the direction of m relative to H is reflected in Ae, one can see in Eq. ([55| that the 
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FIG. 7; (Color online) The IFli^i)!^ curves obtained numerically in H || c case with m ± H (a) and with m || H (b), where 
solid (dotted) curves correspond to the case with (without) the additional internal magnetic field brought by the PPB-induced 
AFM fluctuation. The form factor |-Fi,iP is normalized by {4tvCgi^)^. Highest (red), middle (blue), and lowest (green) curves 
are the |_Fi,i(/i)p ones obtained at t = 0.02, t = 0.06, and t = 0.1 in the inset in Fig. \5\ respectively. Regarding the parameters 
relevant to the AFM fluctuation, ^n = 0.6^0 with £,oPf ~ 7.0 and Sic = are used and hcp = 0.289 is assumed so that the 
location of the AFM instability field is below -ffc2(0). In the high-field and low-temperature region, VLFF is enhanced by the 
PPB-induced AFM critical fiuctuation. 



quasiparticle damping is more efTective in m _L H than in m || H. In examining effects of the AFM fluctuation, we 
take the quasiparticle damping into account with the substitution e„ — >■ e„ + sgn(e„) Ini 'S^Ckp, 0) which is equivalent 
to the replacement 



exp[-2a;lml]^(ki.,0)] 



sinh \2TTt : 



sinh[27ri; 



(57) 



in Eqs. ([^5]) . In the evaluation of the amplitude of the SC order parameter Aq, the replacement ((57)) is also used. 

Figure [7] shows the field dependences of VLFF |Fi,ip in H || c case with m _L H (a) and m || H (b), where solid 
and dotted curves correspond to the cases with and without the internal magnetic field brought by the PPB-induced 
AFM fluctuation, respectively, at i = 0.02 [highest (red) curves], t = 0.06 [middle (blue) ones], and t = 0.1 [lowest 
(green) ones]. The AFM instability field hcp is assumed to be located below i?e2(0) so that hcp is consistent with 
both the experimental result suggestive of the AFM critical point below Hc2 (0) [3, [l^, U^ and our theoretical result 
shown in Fig. [SJ In both m _L H and m || H, VLFF at the low temperature t = 0.02 is remarkably enhanced in 
the high-field range by the PPB-induced AFM critical fluctuation, and as a result, the slope of the \Fi^i{h)\'^ curve 
becomes sharper. On the other hand, in m _L H (Fig. [7] (a)), VLFF is rather suppressed by the PPB-induced 
AFM fluctuation at t = 0.06, while, in m || H (Fig. [7] (b)), the enhancement of VLFF due to the AFM fluctuation 
can be seen at the same temperature. To understand the physical origin of the enhancement and the suppression 
of VLFF due to the PPB-induced AFM fluctuation, spatial distributions of the internal field brought by the AFM 
fiuctuation bg^^. (r) in m || H and m _L H at i = 0.06 and h = 0.25 are shown in Fig. [8](b) and (c), respectively. For 

comparison, the internal field without the AFM contribution b^ (r) 4- b^ (r) at the same temperature and field is 
also shown in Fig. [S](a). In Fig. [51 the contributions from the six Fourier components with the shortest jKj^j, ±(2, 0), 
±(1,1), and ±(1,— 1), are extracted from the full summation for Kj^ so that one can easily grasp the tendency of 
the flux distribution. Although, as shown in the inset in (a), b^ (r) -I- b^ (r) with the full summation for Kj^ has 
a short length scale structure due to relatively large higher Fourier components, the tendency of the distribution is 
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FIG. 8: (Color online) Spatial distributions of the longitudinal internal magnetic field normalized by inCcL without the AFM 



,(4) 



(2,2)/ 



,(2,2), 



contribution b^ (f) + b^ (f) (a), the internal field brought by the AFM fluctuation bjjj^ (f) in m || H (b), and bjjj^ (f) in 
m _L H (c) at i = 0.06 and h = 0.25 in the case with aM,c = 5.8, where dominant contributions coming from the Fourier 
components ±(2, 0), ±(1, 1), and ±( — 1, 1) are extracted from the full internal fields. The inset of (a) shows the full internal field 
b^ +b^ . Although higher Fourier components become relatively large at low temperatures and express the flux distribution 
at short length scales, the tendency of the distribution is well described by the six components listed above. Enhancement of 
the magnetic flux in the vortex core region and its reduction outside the vortex core are commonly seen in both b^ (f ) + b^ (f ) 



,(2,2), 



,(2,2) 



and bjj^'^ (f) in m || H, while, in b^^^ (f) in m _L H, the magnetic flux is enhanced outside the vortex core. The difference in 
^fluc (^) ^^ t*° cases, m II H and m _L H, is reflected in VLFF shown in Fig. [T] 



well described by the six components listed above. As one can see in Fig. [51 enhancement of the magnetic flux in the 



(2), 



,(4)/ 



vortex core region and its reduction outside the vortex core are commonly seen in both b]^'(r) + b^'^(r) and b^J^'(r) 

("2 2) 

in m II H, while, in bfj^j'^'(r) in m _L H, the magnetic flux is enhanced outside the vortex core, in other words, the 
magnetic screening outside the vortex core is suppressed. This result suggests that the enhancement (suppression) 
of VLFF due to the AFM fluctuation corresponds to the promotion (reduction) of the magnetic screening outside 
the vortex core. Further, comparing Fig. [7] (a) with Fig. [S] we can see that it depends on a sign of the SC-AFM 
coupling term T^ ' ^ 

j'2 2^ ('2 2^ 

high-field and low-temperature region where -F)^^ ^ is negative, bg^'^ (r) enhances VLFF, while, in lower-field and 



(2 2) 

whether the AFM contribution b^^^ (r) promotes the magnetic screening or prevents it. In the 



r(2,2) 



Ao,m ^° iic&ciuivc, i^g^^ 

higher-temperature region where -^/^'^m ^^ positive, bg^^^(r) suppresses VLFF. Noting that, from the viewpoint of the 

(2 2) 

SC order, negative values of J-)^ ' ^ indicate that the stability of the SC order is enhanced by the AFM fluctuation, 
we can understand the mechanism of the enhancement of VLFF due to the PPB-induced AFM fluctuation as follows: 
the SC state becomes more stable due to the PPB-induced AFM fluctuation, then the SC magnetic screening is 
promoted, and as a result, VLFF is enhanced by the internal magnetic field originating from the promoted screening 
current. The falling-down behavior toward the Hc2 transition in the |-Fi.i(/i)p curves is more remarkable in m _L H 
than in m II H, which is due to the fact that the amplitude of the SC energy gap is much suppressed by the stronger 
quasiparticle damping in m _L H. 

The increasing behavior in the field dependence of VLFF becomes remarkable due to the enhanced magnetic 
screening caused by the PPB-induced AFM fluctuation. Further, the obtained field dependences of VLFF in lowering 
temperature in both m _L H and m || H, are consistent with the experimental data on CeCoIns [lj|. Considering 
these fact, we could conclude that the strong AFM fluctuation around Hc2(0) observed in CeCoIns in H || c is the 
PPB-induced AFM fluctuation enhanced strongly by increasing field and decreasing temperature and that the abrupt 
increase of VLFF at high fields is a consequence of the magnetic screening enhanced by the PPB-induced AFM critical 
fluctuation. 

Concerning the direction of the AFM moment vector m, we cannot rule out alternative possibilities of m _L H or 
m II H by comparing our theoretical result with the experimental data. However, from the experimental data showing 
that the configuration m || c is realized in H || ab [2^, [2y|, it is inferred that m is locked along c-axis. Further, noting 
that the AFM fluctuation in the normal state is suppressed by the Zeeman effect in m || H while it is not affected by 
the Zeeman effect in m _L H, the absence of the AFM order in the SC phase in H || c seems to be understood as a 
result of the suppression of the AFM fluctuation with m || c in the normal state. 
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IV. ANTIFERROMAGNETIC ORDER IN H |1 ab 

In this section, we discuss a possible AFM order realized in vortex states in the d-wave superconductor with 
strong PPB. Throughout this section, we assume that the AFM moment vector is perpendicular to the magnetic field 
m _L H II ab, since the configuration m || c has been established in the AFM order observed in the HFLT phase of 
CeCoIus in H || ab case. In the parallel field, strong PPB induces the longitudinal FFLO vortex lattice state in the 
HFLT corner in the phase diagram 35] and the FFLO region is large so that it may not be ignored. Although the 
FFLO state is not taken into account in the analysis on the PPB-induced AFM order in the SC vortex lattice, we 
will discuss eflFects of FFLO spatial structures on the AFM order and also on VLFF later. 

A. Modulated antiferromagnetic order in the SC vortex lattice 

Below, we consider the situation where the PPB-induced AFM fluctuation becomes so strong that an AFM order 
appears inside the SC state, i.e., (Jvn + -^a ,„)/"^^ < 0. Because the SC gap function has the vortex-lattice 
modulation, an AFM order with a spatial modulation synchronized with the vortex lattice is expected to appear 
instead of the spatially uniform AFM order. Possible modulated AFM orders are classified into two types: the AFM 
order localized in the normal-state region, namely, in vortex cores and the one coexistent with the SC order. Since 
we consider the spatial distribution of the PPB-induced AFM order in the presence of a potential brought by the SC 
vortex lattice through the SC-AFM coupling term -^^'m ' ^^ ^^ natural to assume that m{r) takes the following form: 

7r,(XY)= "" M -J- ,/, y ' T^^^ exp [iK_L • (X, r)l ) , (58) 



yr,2 Y^'_ ,„|t(2) |2 



E'^.MWJ'^ K^^O 






or equivalently. 



.(2) 



^(q) = , === '^q.o + w T^[ <5^ ji^^o ) , (59) 



where w is a variational parameter to be determined by minimizing {J-m + -7-"^' m)/"^^' ^'^'^ X^k m denotes the 
Kj^-summation only for the lowest Fourier components, {mi, 1112) = ±(1,1), ±(1,— 1), and ±(2,0). We believe that 
the restricted summation for Kj^ gives correct results because the contribution from higher Fourier components will 
be negligibly small and will not affect the results. It should be noted that negative (positive) values of w indicate the 
AFM order enhanced (suppressed) in the vortex core. 

In obtaining spatial structures of AFM orders, we assume that the magnetic flux is uniform. Since, in general, 
an inhomogeneous internal field is induced by a spatially modulated AFM order, the induced inhomogeneous field 
may affect the spatial structure of the AFM order. In this study, however, we neglect the internal field brought by 
the AFM order for the following two reasons: First, the leading order AFM contribution bm' (r) vanishes because 
of the configuration m ± H and does not affect the magnetic fiux at all. Second, the internal field brought by the 
AFM order thorough the SC-AFM coupling b2^^'J(r) is considered to be negligibly small since, as we will see later, 
the transition to the AFM order occurs at a relatively high field so that the amplitude of the AFM order parameter 
will not grow up to a large value with increasing field. 

Figure |9] shows an example of the temperature- field phase diagram of the PPB-induced AFM order inside the d-wave 
SC vortex lattice state in H || ab, where a thin (red) solid curve and a dashed (red) one denote the transition curve to 
the AFM order with a vortex-lattice modulation and the one to the spatially uniform AFM order, respectively, and 
both of them are second order transition curves. As one can see in the main panel of Fig. [51 the stability region of 
the PPB-induced AFM order is expanded by the modulation synchronized with the vortex lattice, and an example 
of the spatial structure of the modulated AFM order is shown in the inset of Fig. ^ As we will discuss later, the 
PPB-induced AFM order is not localized in the vortex core but coexistent with the SC order. Figures [TO] (a) and (b) 
show the field dependence of J-m +-^a ' m ^^^ ^^^^ "^^ ^^^ variational parameter w at t = 0.03 in Fig. [SJ respectively. 
For comparison, results in the case with ajv/.ab = 0.01 are shown in the figures as dotted (green) curves. Figure [TOl 
suggests two important results. First, the AFM order with the vortex-lattice modulation becomes more stable in the 
vortex state than the uniform one. Second, as clearly seen in Fig. [TO] (a), in the case with aM,ab = 7.5, the AFM 
order appearing inside the SC state lowers the free energy, while, in the case with aM,ab = 0.01, it never lowers the 
free energy. In the case with weak PPB, the conventional competitive nature between SC and AFM orders is reflected 
in the free energy cost and negative values of w, which suggests that, in systems with a sufficiently large T^/Tc in 
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FIG. 9: (Color online) Example of the numerically obtained temperature-field phase diagram of the PPB-induced AFM order 
inside the d-wave SC vortex lattice state in H || a6 case, where a thick (black) solid curve and a thin (red) solid one denote the 
first order Iic2 transition and the second order transition to the AFM order with a spatial modulation commensurate with the 
vortex lattice structure, respectively. For comparison, the onset of the spatially uniform AFM order in the SC vortex lattice 
is also shown as a dashed (red) curve. The parameters used in obtaining the phase diagram are QM.ab = 7.5, 5ic = 0.001, 
and T-^/Tc = 0.027. The PPB-induced AFM order becomes more stable in the vortex state with the spatial modulation 
commensurate with the vortex-lattice structure. The inset shows the spatial profiles of the SC gap |A(a;)p (dashed curve) 
and the modulated AFM order |m(a:)p [solid (red) one] along the trajectory x between neighboring vortex centers. The PPB- 
induced AFM order realized in the d-wave SC vortex state is not localized in the vortex core but coexistent with the SC order. 
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10: (Color online) Field dependences of Tm' + J^/^ ' „ 



' ■^^ ' ' (a) and the variational parameter w (b) at f = 0.03 in Fig. [O] For 
0.01 are also shown in the figures as dotted (green) curves. In (a), one can see 



comparison, results in the case with aM,ab 
that the occurrence of the AFM order in the high-field SC phase lowers the free energy in the case with strong PPB, while, in 
the case with weak PPB, it never lowers the free energy. Further, as shown in (b), the optimized value of w is positive in the 
case with strong PPB, suggesting the AFM order coexistent with the SC order, while, in the case with weak PPB, the familiar 
competition between the two orders is indicated by negative values of w. 



zero field, the AFM order localized in the vortex cores, i.e., in the normal-state region where |A| = can appear 
[43 . |44| as shown in Fig. [TT] (c). In contrast, as denoted by a solid curve in Fig. [TO] (b), optimized values of the 
variational parameter in the case with strong PPB are positive, indicating the AFM order coexistent with the SC 
order. An example of the spatial distribution of this PPB-induced AFM order is shown in Fig. [TT] (b), where the 
spatial structure of the vortex lattice is shown in Fig. [TT] (a). The AFM order is suppressed in the vortex core and 
enhanced in the region with large |A|. The profiles of the SC gap |A(2:)p and the PPB-induced AFM order |TO(a;)p 
along the trajectory x between neighboring vortex centers are shown in the inset of Fig. ^ 

Although we have used a small value for 6ic in the above numerical calculation, the PPB-induced AFM order 
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FIG. 11: (Color online) Spatial structures of the SC gap |A(f)p (a) and the modulated AFM order |77i(f)p in the cases with 
OiM,ab = 7.5 (b) and aM,ab = 0.01 (c) at i = 0.03 and H/Hc2,ab{0) = 0.75 in Fig. 1101 The coordinates x and z are dimensionless 
and defined as a; = -f'^'^x/rH and z = 'y^''^z/rH, respectively. The figures (b) and (c) correspond to w = 0.63 and w — —0.15, 
respectively. In the case with strong PPB, the AFM order coexists with the SC order while, in the case with weak PPB, the 
AFM order competes with the SC order and is locahzed in the vortex cores. 



becomes much stable in the SC phase as |i5ic| is increased Q- On the other hand, the incommensurate wave vector 
observed in the AFM order appearing inside the HFLT phase of CeCoIns [2^, [2g| seems to originate from the deviation 
from the perfect nesting condition of the Fermi surface [301, which suggests that the deviation \Sic\ is large enough 
and the momentum dependence in \5ic\ should be taken into account in the detailed discussion on the AFM order in 
the HFLT SC phase. 

Our result obtained here for the PPB-induced AFM order in the SC vortex lattice is consistent with recent NMR 
data suggesting that the AFM order in the HFLT phase of CeCoIns is spatially extended without being localized in 
the normal-state region |22,] . Although the longitudinal FFLO vortex lattice state is ignored in the above calculation, 
the theoretical study taking account of the longitudinal FFLO modulation along H without in-plane vortex-lattice 
structures included shows that, at least in the high-field side of the FFLO state, the AFM order is not localized in 
the FFLO nodal planes on which |A| — 0, but coexistent with the SC order 130. These results obtained in the two 
approaches, which are complementary to each other, suggest that the PPB-induced AFM order coexistent with the 
SC order may be stabilized at least in the high-field side of the longitudinal FFLO vortex lattice state. 



B. VLFF in the modulated AFM order 



We will briefly discuss effects of the PPB-induced AFM order on VLFF. A red solid (dashed) curve in Fig. [T^] 
denotes the field dependence of VLFF at t = 0.03 in Fig. [9] in the presence (absence) of the PPB-induced AFM 
order modulating with the vortex lattice. Here, the variational parameter w in Eq. (j58p is determined by minimizing 
J"gl(Ao,to) — J^gl(Ao,0), while the amplitude of the AFM order parameter |7tt,| is obtained by using Eq. ([T^. The 
form factor is enhanced by the PPB-induced AFM order as a result of the additional magnetic screening brought 
by the occurrence of the PPB-induced AFM order, like in the case only with the PPB-induced AFM fluctuation 
examined in Sec. HI. However, the enhancement of VLFF brought by the AFM order is quite small compared with 
that brought by the AFM critical fiuctuation. We expect an additional AFM fluctuation around the nonvanishing 
AFM order parameter, which has not been taken into account in obtaining Fig. 1121 to further enhance VLFF. 



C. Internal magnetic field in the longitudinal FFLO vortex lattice state 



So far, we have not examined effects of the longitudinal FFLO structure of the SC order parameter on the internal 
fleld and VLFF which should be seen in the HFLT corner in the phase diagram [3a|. Below, we examine the flux 
distribution in the longitudinal FFLO vortex lattice although, in turn, an AFM order will be neglected for brevity in 
contrast to the case of Fig.l2. 

In the longitudinal FFLO vortex lattice state, the SC gap function is expressed in the form 



A{r) ^ Aipo{X,Y) V2COS {Q Z) 



(60) 



with the modulation wave vector QZ parallel to the magnetic fleld. This FFLO modulation in the SC order parameter 
affects the internal fleld through the operator acting on the SC gap function exp [iAvp • II] A(r) — \/2Aexp [iAvp • 
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FIG. 12: (Color online) Field dependence of VLFF |-Fi,ip at t — 0.03 in Fig. [9] obtained in the mean-field approximation, 
where |i^i,i| is normalized by (IttCgl) • A red solid (dashed) curve corresponds to the result in the presence (absence) of 
the PPB-induced AFM order with the vortex-lattice modulation. Although VLFF is enhanced by the occurrence of the AFM 
order, the enhancement is not so remarkable compared with that brought by the PPB-induced AFM fiuctuation. The |-Fi,i(ft)| 
curves including the longitudinal FFLO spatial variation of the SC order parameter ignored here are shown in Fig. 1131 



Uj_]ipq{X,Y) exp [iA{vp ■ Z){-iVz)] cos(QZ), and then, the MaxweU equation -V x h^ (r)/{ATrCGL) = V x 
(Ek^sJO ['''^K.l Q + "^K_L Q^^^ (2(5Z)]e'^^'^'^'^^).Z is obtained. It can be shown that the Maxwell equation has 



a solution with a transverse component in b^^(r) [36 
expressed in the form 



The resulting 0(|Ap) contribution to the internal field is 
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FS 



As one can see in Eq. (|6ip . the transverse magnetic field is induced by the finite FFLO modulation {Q ^ 0) and the 
longitudinal magnetic flux varies along the FFLO modulation. An example of the flux distribution in the longitudinal 
FFLO vortex lattice is shown in Fig. [T3] (a). One can see that outgoing and incoming fields are induced and 
correspondingly, the longitudinal magnetic flux varies along the FFLO modulation. The experimentally measured 
VLFF is expected to reflect the internal fleld which is spatially averaged over the range [—tt/Q, tt/Q] in the Z direction. 
Then, the transverse incoming and outgoing fields cancel each other. The jAj"* contribution to the spatially averaged 

internal magnetic field is given by replacing bj^' ^ (pj) in Eq. (H^ by b\^ ^ (pj) [cos (Q (vp • Z) {aj — jSj — 7^ -j- 

5j)) /2 -I- cos (Q (vp • Z){aj + Pj)) cos {Q (vp • Z)(7j -I- 5^))] . Figure [H (b) shows typical |Fi,i(/i)p curves in the 
longitudinal FFLO vortex lattice without an AFM order, where solid (blue) and dotted curves correspond to the 
results with and without the FFLO longitudinal modulation, respectively, and the arrow denotes the FFLO transition 
field. In obtaining the result, the modulation Q is determined selfconsistently from the GL free energy functional 
[35| , and further, it is confirmed that the FFLO transition is of second order. As one can see in the figure, VLFF 
is suppressed by the longitudinal FFLO modulation and the slope of the \Fi^i{h)\^ curve changes its sign quite close 
to the FFLO transition field. An abrupt falling down behavior in VLFF has been experimentally observed near the 
transition between the low field SC phase and the HFLT one in H || ab case [43], which seems to imply the formation of 
the longitudinal FFLO vortex lattice state in the HFLT phase. On the other hand, an AFM order and corresponding 
AFM fluctuation which are not taken into account in the above argument about the FFLO state may affect the result. 
It should be noted, however, that the quasiparticle damping brought by the AFM critical fiuctuation should suppress 
the PPB effect so that the onset of the FFLO transition field in the above calculation should be shifted to a higher 
field and that the FFLO region should be narrower. In order to investigate the high-field and low-temperature SC 
phase in H \\ ab, further studies taking account of the longitudinal FFLO vortex lattice, the PPB-induced AFM order, 
and fluctuation around the AFM transition consistently will be needed. 

In the last of this subsection, we will discuss effects of the longitudinal FFLO modulation on VLFF in H || c case. 
The HFLT phase appearing just below the Hc2{T) curve as a narrow region in H |j c case [lallJI has been considered 
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FIG. 13: (Color online) Spatial flux distribution (a) and the |-Fi,i(/i)| curves normalized by (47rCGL) (b) in the longitudinal 
FFLO vortex lattice state. The AFM order and fluctuation are not taken into account in obtaining these results. The structure 
of this state is schematically shown in (a) where a solid (black) vertical line, a vertical wavy (green) curve, and horizontal dashed 
lines indicate a vortex line extending along the applied field, the FFLO modulation, and the nodal planes on which |A| =0, 
respectively. The left upper (lower) panel in (a) shows the numerically obtained transverse field on the surface QZ — ir/A 
{QZ — — vr/4) in the FFLO state with 27r^oQ ~ 0.55 appearing at t = 0.1 and h = 0.65, where a (black) dot denotes the vortex 
center. An outgoing field from the vortex center is seen at QZ = 7r/4, while, on the surface QZ = — 7r/4, rather an incoming 
field is induced. Correspondingly, the longitudinal magnetic flux at the vortex core denoted by vertical (blue) arrows varies 
along the applied field with its minimum at the nodal plane and maximum at the surface on which |A| is the largest. In (b), 
(blue) solid and (blue and black) dotted curves denote the |_Fi,i(/i)p ones with and without the longitudinal FFLO modulation 
at t = 0.1, respectively, and the arrow denotes the corresponding second order transition field to the longitudinal FFLO vortex 
state. The VLFF is suppressed due to the FFLO spatial modulation of the SC order parameter. 



to be a realization of the longitudinal FFLO vortex lattice [13] ■ On the other hand, VLFF obtained in the neutron 
scattering experiment does not show any structure around the transition between the HFLT phase and the lower field 
phase [14], in contrast to the abrupt falling down behavior in H || a6 case discussed above. The absence of a precursor 
of the FFLO state in VLFF in H || c case may be due to the fact that the effect of the FFLO modulation on VLFF is 
too small just above the FFLO transition to be seen. Although VLFF should be suppressed by the FFLO modulation 
Q, drastic suppression cannot be seen because, in H || c case, the stability region of the FFLO state is quite narrow 
and Q cannot grow up to a large value. We believe that, if VLFF is detected in sufficiently high resolution, the onset 
of the FFLO transition should be observed in VLFF in H 11 c case. 



SUMMARY 



Antiferromagnetic (AFM) fluctuation and order induced inside a d-wave superconducting (SC) phase by strong 
Pauli-paramagnetic pair-breaking (PPB) [^ have been theoretically studied, focusing on how they are affected by 
spatial modulations of a SC vortex lattice. Based on the microscopic calculation taking account of both orbital and 
paramagnetic pair-breaking effects, we have shown that, with increasing field and decreasing temperature, the SC 
vortex lattice state becomes more stable by inducing AFM fluctuation inside it: From the viewpoint of the SC order, 
the magnetic screening is promoted by the induced AFM fluctuation in the high-field SC phase. In other words, the 
AFM order can more easily appear inside the SC vortex lattice state than in the normal state. These implications have 
been concretely discussed in connection with the anomalous SC phenomena observed in CeCoIns. First, in H || c case 
where the AFM quantum critical behavior around i/c2(0) is experimentally observed, such appearance of the AFM 
fluctuation in the SC state in higher fields has been explained as a result of the PPB enhanced with increasing the 
field. Then, it is found that the vortex lattice form factor (VLFF) is enhanced by the additional magnetic screening 
brought by the coupling between the induced AFM fluctuation and the spatial modulation of the SC vortex lattice 
and that the VLFF's enhancement becomes much remarkable as the AFM instability is app roached. The obtained 
temperature and field dependences of VLFF are consistent with the experimental result [Ij, |lj| , which suggests the 
anomalous field dependence of VLFF observed in the experiments is a consequence of the PPB-induced AFM critical 
fluctuation. Second, in H || ab case where an AFM order with m _L H appears inside the SC phase in CeCoIns, it is 
found that the vortex-lattice modulation promotes the AFM fluctuation and that, as a result, a spatially modulated 
AFM order synchronized with the vortex lattice is stabilized in the SC vortex state. In contrast to the conventional 



23 

competitive nature between the AFM and SC orders, the PPB-induced AFM order in the vortex lattice coexists with 
the nonvanishing SC order parameter and does not localize in the vortex cores. The spatial modulation peculiar to 
the longitudinal FFLO state, which seems to be realized in the HFLT phase of CeCoIn5 [23|; has been neglected in our 
analysis on the AFM order. According to the theoretical study taking account of the longitudinal FFLO modulation 
in the Pauli limit without vortex lattice structures included jiOj, a possible PPB-induced AFM order in the high- 
field side of the FFLO state is not localized in the FFLO nodal plane but prefers to coexist with the nonvanishing 
SC order parameter. The unconventional coexistence of the AFM and SC orders, obtained in the two approaches 
complementary to each other, is consistent with a picture on the HFLT phase of CeCoIus suggested from the recent 
NMR data [2^. Further, our study on effects of the longitudinal FFLO spatial modulation and the AFM order 
on VLFF performed additionally has shown that, as well as the AFM fluctuation, the AFM order enhances VLFF, 
while the longitudinal FFLO modulation leads to a significant reduction of VLFF similar to that seen in CeCoIus. 
These results on the magnetic properties in high field superconductors with strong PPB imply that the HFLT state 
of CeCoIus is a coupled phase composed of a PPB-induced AFM order and the d-wave SC order accompanied by the 
longitudinal FFLO modulation. 
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VII. APPENDIX 

A. Orbital pair-breaking effect 

Here, we show the details of the calculation in obtaining Eq. (P^ and derive equations useful in incorporating the 
orbital pair-breaking effect. In the case with H || £ (H || y), the vector potential can be expressed as Aq = (0, Hx, 0) 
(Ao = {Hz, 0,0)), and then, [H2:,Hy] = ~irjj^ {[IIz,Ilx] = —if'^) is satisfied. In H || z case, by introducing the 
operators Hj- = r//(Hj. ± iHj,)/-\/2, we obtain 



exp 



iAT^Vp-n 



^ g--^A^gi^'?*n+giA,,n_ 



giAjj*n+ ^ g(iAj)*r„/2)^[n,,jn„] g-(Aj)*rH/\/2)n^ gi(A77*rH/V2)n, 



(63) 



where the operator identity e'"^'^! = e"'^'^'/^ e^ e^ is used. In H || y case, (H^;, Hj,) is replaced by (7^^/^nz, 7^/^na;) 
in Eq. (|63|) . Equation (|29| is obtained bv using Eg. ((63)) . Since the Fourier transformation of (po( AT + Q;r//, F)(/3q (AT -|- 
j3rH,Y) with the reciprocal lattice vector Kj^ = {Kx,Ky) is calculated as 



dXdY 



■'Po 



(A + arff,r)V55(A + /3r^,y)e-'K^-(^'^) -T^'' e-3("-«'exp 



'■pl- + ^K:,rH .eveJ i^^-.integer ^°^ \ 2 k 



TT KyVH 



k IT 



exp 



LxLy 

we obtain 

exp is^^T^^ I Vp • Hi a + Vp • H2 /3 



i^-—{a + l3) 



exp 



KyTH 



-^IK^^i/P 



[a - p)\ , 
(64) 



exp 



]{X^ 



V2 



{a- l3)+i 



.{ri X Kx_rH)z 



V2 



(a + /3)) 



(65) 



In the same manner, the identity corresponding to the fourth order term is given by 



exp 



i Se„T-^ (vp • Hi a + vp • n^ ^ + vp • Ha 7 + vp • nj 6^ 



ipo{si)ip*Q{s2)ipis3)iPo{Si)\s,^(^X,Y) 
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cxp 



X 

xcxp 
xexp 



OO 



cxp 



i((a-7)V^ + (/3-'5)V) 



K 



/l ,^2 — ~00 



k k 



^)W-s) 



'(q-Kj^rn) 



T, 



(4) _ r 



2\/2 



K 



':K, 



rH + ^^ 



„. exp 



. (J7 X K^rg), 

■n KyTH (k ^ ^ ^ ^y^/f 
4 /fc 



Vtt 



exp 



\K^rH\ 



T^^){l,,l,,K^)=cos(^i^K,rH + ^^){h+l2) 



xexp 
where Zi and I2 are integers. 



^J(('l- ^2)^-^1+^2 + ^)^ 



exp 



'^{ih-hr + ih+h+^f 



,(66) 



Since the GL coefficients Vi in Eq. 
expressions of Vi which are given by 



B. GL coefficients Vi in Eq. ^ 

dl]) have been already derived elsewhere [23, [33], we only show the final 



where 



1 f°" 

V2 = -Hh)+J dp 



■ cxp 






/cos(p,p)<^|wppexp(^ 



2 '^ / / FS 



Wpl exp 
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1/4= = 2jJ / dp, /cos(X!/'»'X!^ 
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C0s(/6 



FS 



^2\r„2 I / „ n2i 



i?24 = Re(r;^)[p^ + (p3-pi)1, 
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Im(r7^) 
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P3 + ^4 + (P2 - P5)^ 

Pi + 2(p3 +P4 + P5), 



65 == P2 - P3 - P4 - P5, 

66 == (P4 - Pbf + (Pl + P5 - P3)^ 

67 = (Pl +P4-P3)^, 



(67) 
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68 = (P3 - Pa? + {P2 + P3- Pbf, 

eg = (P2 +/04-p5)^- (68) 

C. Imaginary part of the self energy 

The self energy of a quasiparticle is expressed as [33| 

E,(k,ze„) - T^^g,,,^(k + Qo-q)x(q,i(£«-ei))- (69) 

£1 q 

After carrying out an analytic continuation, we obtain the imaginary part of the self energy on the Fermi surface as 
follows: 



1 /'°° 

ImE^(k,0) = ttY. dxIm^^,,-(k + Qo-q)Imx^(q,x 



4(27r)2 7 27r7 ^^ ^ V ^ ■->' ^^ V iV(0)eS 1 + ^^(/i, 9^ 



coth I — I — tanh ( — | 
\2TJ \2TJ 

^^{h,t) 2T 

F2 



(70) 



where S^ is the retarded self energy, Q^ is the retarded Green's function, and the inequality \x\/2T ^ 1 is used. 
Since the (5-function in Eq. (jTO)) is reduced to 

<r.<Jic+v..q-(^+.)/T.) ^ %^-[Ae/Y) ^(^„^„) (71) 

V / avp\ smffln 



with 00 = cos ^ (Ae/{qvF)), we obtair 
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2 iv(o)eS(2^)2T;^7(^^/„^). y?~(AiA;;o^c-2(/i,i) + 'z' 
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.(^'(^0 + (A£/^^i.)^) . (72) 
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